Skip to content

feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is allowed - #9754

Draft
mhk197 wants to merge 2 commits into
mk/bitpacked-v2from
mk/bitpacked-v2-editions
Draft

feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is allowed#9754
mhk197 wants to merge 2 commits into
mk/bitpacked-v2from
mk/bitpacked-v2-editions

Conversation

@mhk197

@mhk197 mhk197 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked on #9750. Lets the writer emit fastlanes.bitpacked_v2 when its editions permit it. A compressor scheme looks at what is enabled and produces the newest available format.

  • CascadingCompressor::with_allowed_serialized_ids / allows_serialized_id. The compressor carries the serialized IDs its output may use. The file writer passes the same set it gives the array context, right where it already filters schemes by allowed encodings, and the compressor knows nothing about editions themselves. Without a restriction, which is what BtrBlocksCompressor::default() gives in-memory callers, every ID is allowed and the newest format wins.
  • BitPackingScheme produces the newest allowed format. It picks a width per 1024-element chunk when fastlanes.bitpacked_v2 is allowed and one global width otherwise, so an old edition keeps writing the original format and a newer one gets per-chunk widths from the same scheme. There is no second scheme and no per-scheme wire-ID declaration. The width table child is re-encoded through compress_child, like the children of the decimal and temporal schemes.
  • Mostly-patched arrays stay primitive. The scheme returns the original array when half or more of the values would be patches. A few wide values otherwise pack at width 0 with nearly everything patched, which beats raw storage by a couple of buffer bytes and loses them back in footer metadata.
  • The CUDA preset allows only each encoding's original format, whose ID is the encoding's own, since CUDA has no per-chunk bit-unpacking kernel yet.
  • No shipped edition permits the format yet. A preview entry signals near-certainty about the wire format, and the format was still moving this week. The edition entry can follow once it is blessed; the writer test declares a test-local edition instead.

Tests

  • core_writer_never_emits_bitpacked_v2: a session with the default encodings and only the core edition enabled writes a column whose chunks need 1 to 22 bits, and every array it produces reads back as fastlanes.bitpacked.
  • permitting_writer_emits_bitpacked_v2: the same session plus a test-local edition permitting the format writes the column as fastlanes.bitpacked_v2 and it reads back equal.
  • cuda_compatible_disallows_per_chunk_bitpacking: the default compressor allows the format and the CUDA preset does not.
  • test_mostly_patched_stays_primitive: six i64 values, five of them wide, stay primitive instead of becoming width 0 with five patches.
  • The new_array_context tests cover the allowed set: the editions' serialized IDs, or every registered ID when editions are disabled.
  • The vortex-btrblocks golden snapshots show per-chunk widths and a width_table child for bit-packed columns in both the default and unstable configurations; the four default goldens that change all shrink.

Validation

  • cargo nextest run -p vortex-compressor -p vortex-file -p vortex-edition -p vortex-btrblocks -p vortex editions, with and without --features unstable_encodings
  • cargo clippy --all-targets --all-features on vortex-compressor, vortex-file, vortex-edition, vortex-btrblocks, and vortex with unstable_encodings; cargo +nightly fmt --all

@mhk197
mhk197 force-pushed the mk/bitpacked-v2-editions branch from 5239669 to f3fd087 Compare September 3, 2026 15:40
@codspeed-hq

codspeed-hq Bot commented Sep 3, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 2338 untouched benchmarks
⏩ 206 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime arrow_checked_add_u32_avx512[16384] 17.6 µs 21.3 µs -17.1%
WallTime arrow_checked_add_u32_neon[16384] 20.4 µs 12.7 µs +59.84%
WallTime mul_i32_nullable_avx512 9.6 µs 8.7 µs +10.49%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/bitpacked-v2-editions (114a0f0) with mk/bitpacked-v2 (dcd7e85)

Open in CodSpeed

Footnotes

  1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mhk197
mhk197 force-pushed the mk/bitpacked-v2-editions branch 2 times, most recently from b978a04 to 9d950f7 Compare September 3, 2026 16:38
@mhk197
mhk197 force-pushed the mk/bitpacked-v2-editions branch from 9d950f7 to 01b289f Compare September 3, 2026 18:52
@mhk197
mhk197 force-pushed the mk/bitpacked-v2-editions branch from 01b289f to 6258f9d Compare September 3, 2026 19:37
Comment thread docs/specs/editions.md Outdated

#### `preview2026.09.0`

- `array`: `fastlanes.bitpacked_v2`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a caution that preview means that we are 99% certain that this is the right serialisation format

Comment thread vortex-compressor/src/scheme/mod.rs Outdated
/// common case of an encoding with a single wire format. A scheme whose output requires a
/// newer format of an encoding lists that format here, so a writer restricted to editions
/// without it drops the scheme instead of producing arrays it cannot serialize.
fn produced_serialized_ids(&self) -> Vec<ArrayId> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joseph-isaacs and I convinced ourselves that this is unnecessary. The logic is that in old edition there's only old compressor, in new edition there's the new compressor. Then upon serialisation on old edition you will produce bitpacked v1 because that's what the compressor produced (i.e. you can unwrap v2 to v1) and in the new edition you will produce v2. If unwrapping is not possible it's a different encoding.

I think we need a better way to configure the compressor...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here its not always possible to downgrade v2 -> v1 though right? if there are different bit widths per chunk

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually nvm, if new edition is disabled an array written with v1 has to round trip (disk) v1 -> (mem) v2 -> (disk) v1 unless there's been some intermediate transformation

@robert3005

Copy link
Copy Markdown
Contributor

I think you want single compressor scheme if you have single in memory array. Then if v2 is allowed we produce v2, otherwise produce v1

@mhk197
mhk197 force-pushed the mk/bitpacked-v2-editions branch from 6258f9d to 9f9fc0c Compare September 3, 2026 21:16
@mhk197 mhk197 changed the title feat(fastlanes): put per-chunk bit widths behind a preview edition feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is permitted Sep 3, 2026
CascadingCompressor carries the serialized IDs its output may use. The file
writer passes the same set it gives the array context, next to the existing
filter on allowed encodings, so a scheme whose encoding has several wire
formats can produce the newest one still allowed without the compressor
knowing about editions. The CUDA preset allows only each encoding's original
format, since it has no per-chunk bit-unpacking kernel yet.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…2 is allowed

BitPackingScheme chooses a width per 1024-element chunk when the compressor
may use the v2 format and one global width otherwise, so an old edition keeps
writing the original format and a newer one gets per-chunk widths from the
same scheme. The width table child is re-encoded through the cascade.

The scheme returns the original array when half or more of the values would
be patches. A few wide values otherwise pack at width 0 with nearly every
value patched, which beats raw storage by a couple of buffer bytes and loses
them back in footer metadata.

No shipped edition permits the format yet; the writer test declares its own.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 changed the title feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is permitted feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is allowed Sep 3, 2026
@mhk197
mhk197 force-pushed the mk/bitpacked-v2-editions branch from 9f9fc0c to 114a0f0 Compare September 3, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants